Privately Aligning Language Models with Reinforcement Learning

Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections

Adaptive Mixture Methods Based on Bregman Divergences

We investigate adaptive mixture methods that linearly combine outputs of $m$ constituent filters running in parallel to model a desired signal. We use "Bregman divergences" and obtain certain multiplicative updates to train the linear combination weights under an affine constraint or without any constraints. We use unnormalized relative entropy and relative entropy to define two different Bregman divergences that produce an unnormalized exponentiated gradient update and a normalized exponentiated gradient update on the mixture weights, respectively. We then carry out the mean and the mean-square transient analysis of these adaptive algorithms when they are used to combine outputs of $m$ constituent filters. We illustrate the accuracy of our results and demonstrate the effectiveness of these updates for sparse mixture systems.Comment: Submitted to Digital Signal Processing, Elsevier; IEEE.or

Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe

Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.Comment: ACL 2023 Main Conference (Honorable Mention

Assessment of the requisites of microbiology based infectious disease training under the pressure of consultation needs

<p>Abstract</p> <p>Background</p> <p>Training of infectious disease (ID) specialists is structured on classical clinical microbiology training in Turkey and ID specialists work as clinical microbiologists at the same time. Hence, this study aimed to determine the clinical skills and knowledge required by clinical microbiologists.</p> <p>Methods</p> <p>A cross-sectional study was carried out between June 1, 2010 and September 15, 2010 in 32 ID departments in Turkey. Only patients hospitalized and followed up in the ID departments between January-June 2010 who required consultation with other disciplines were included.</p> <p>Results</p> <p>A total of 605 patients undergoing 1343 consultations were included, with pulmonology, neurology, cardiology, gastroenterology, nephrology, dermatology, haematology, and endocrinology being the most frequent consultation specialties. The consultation patterns were quite similar and were not affected by either the nature of infections or the critical clinical status of ID patients.</p> <p>Conclusions</p> <p>The results of our study show that certain internal medicine subdisciplines such as pulmonology, neurology and dermatology appear to be the principal clinical requisites in the training of ID specialists, rather than internal medicine as a whole.</p

A BOUNDED COMPONENT ANALYSIS APPROACH FOR THE SEPARATION OF CONVOLUTIVE MIXTURES OF DEPENDENT AND INDEPENDENT SOURCES

ABSTRACT Bounded Component Analysis is a new framework for Blind Source Separation problem. It allows separation of both dependent and independent sources under the assumption about the magnitude boundedness of sources. This article proposes a novel Bounded Component Analysis optimization setting for the separation of the convolutive mixtures of sources as an extension of a recent geometric framework introduced for the instantaneous mixing problem. It is shown that the global maximizers of this setting are perfect separators. The article also provides the iterative algorithm corresponding to this setting and the numerical examples to illustrate its performance especially for separating convolutive mixtures of sources that are correlated in both space and time dimensions